Convergence and Sample Complexity of Gradient Methods for the Model-Free Linear–Quadratic Regulator Problem

Model-free reinforcement learning attempts to find an optimal control action for unknown dynamical system by directly searching over the parameter space of controllers. The convergence behavior and statistical properties these approaches are often poorly understood because nonconvex nature underlying optimization problems lack exact gradient computation. In this article, we take a step toward demystifying performance efficiency such methods focusing on standard infinite-horizon linear–quadratic regulator problem continuous-time systems with state-space parameters. We establish exponential stability ordinary differential equation (ODE) that governs gradient-flow dynamics set stabilizing feedback gains show similar result holds descent method arises from forward Euler discretization corresponding ODE. also provide theoretical bounds rate sample complexity random search two-point estimates. prove required simulation time achieving $\epsilon$ -accuracy in model-free setup total number function evaluations both scale as notation="LaTeX">$\log \, (1/\epsilon)$ .